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INTRODUCTION 


1.1 Identification and Purpose of this Specification 


This document is the Engineering Product Specification, Part 
1, for anew memory queued i/o protocol (MQI0). It sets forth 
the characteristics and requirements of the memory queued 1/0 
protocol and presents examples of its advantages and its use. 


1.2 Scope 


The purpose of the memory queued i/o protocol is to serve as 
a performant, efficient, reliable, and resilient method of i/o 
and interprocessor communication. This new protocol is designed 
for use on all new i/o processors, including those for magnetic 
peripherals, communications, cluster controllers, and others, 
which will be developed for use on the Honeywell Megabus, on 
variants of the Megabus, or on future system busses. 


1.3 Applicable Documents 


A list of reference documentation is provided below in 
Appendix I. 


1.4 Definitions 


A list of acronyms and definitions can be found in Appendix 
soe 
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ARCHITECTURAL OVERVIEW 


2.1 %I/0 Dialogue 


Under the memory queued i/o protocol, communications between 
processors function primarily through memory resident control and 
data blocks. At system startup, the initializing processor 
(typically a CPU running system software) sends a block cf data 
called the Initialization Control Block to the target processor. 
Contained in the Initialization Control Block are pointers to the 
key i/o data structures. This block 15 stored in the controller. 


During normal system operation, the initiating processor 
starts an i/o operation by constructing one or more blocks of 
data in memory. The first of these blocks is called a Request 
Header, and contains i7o control information. It also contains a 
pointer to a queue of subsidiary blocks of data called Request 
Bodies. These Request Bodies describe the particular functions 
that the target processor (typically an i/o controller) is to 
rerform, and also contain pointers to data buffers. When the 
Request Header has been built and the necessary Request Bodies 
have been Built and attached to it, the requesting processor 
attaches the Request Header to the target processor‘s Request 
Queue. Figure 1A describes the situation at the start ofa 
simple i/o operation. 


At this point, it is the target processor‘s turn to act. It 


scans its Request Queue looking for requests. When it discovers 
the new Request Header, it removes it from the Request Queue and 
puts it on the Action Queue for safekeeping. Starting with the 


first Request Body, the processor then works its way sequentially 
through the queue of Request Bodies performing the requested 
functions. Figure 1B illustrates the data structure 
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relationships while i/o is in progress. 


When all functions are complete or when an abnormal 
termination event occurs, the target processor sets status 
information in the Reguest Header, removes it from the Action 


Queue, and attaches it to the Completion Queue. The initiating 
processor finds the Request Header on the Completion Queue. 
Figure 1C illustrates this situation. Removing the Request 


Header from the Completion Queue, the initiating processor 
analyzes the information, thus completing the transaction. 


In the case of unsolicited events, such as device attention 
interrupts, there is a fourth group of data structures called the 
Free Request Queues. These contain ready to use Request Headers, 
complete with linked Request Bodies. The signalling processor 
deqgueues a Request Header (the choice of Free Request Queue to 
pick it from is discussed later), puts it on the Action Queue, 
fills in a few appropriate fields, places the received data, if 
any, into the data buffers, and attaches the whole thing to the 
Completion Queue. 


2.2 Memory Address Handling 


A second important feature of the memory queued i/o protocol 
lies in the handling of memory addresses. Since this protocol is 
designed to be used in aA virtual memory environment, it will 
often be convenient to pass virtual rather than physical 
addresses to the target processor. In order to do this, virtual- 
to-physical address mapping information must be made available to 
the target. This is accomplished by placing copies of several 
segment descriptors in each Request Header. It becomes the 
target processors responsibility to handle the access checking 
and memory mapping necessary to resolve virtual addresses. 


2.3 Comparison with Direct Interface 1/0 


The memory queued i/o protocol resolves several efficiency, 
performance, and integrity problems that have developed as old 
direct interface i/o (DILIO) WAS applied to large and 
multiprocessor systems. With direct interface i/o, CPU-to- 
controller control communication was direct rather than through 
memory. If a controller could not keep up with the data from the 
CPU, it had to reject (NAK) the transfer, which forced the driver 
software to reissue the command, or hold up the CPU (WAIT), 


momentarily stopping the entire bus. If two or more CPUs 
attempted to reach the controller simultaneously, the service 
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delays got bad enough on some configurations that the system‘s 
hang protection timers would expire, causing traps and other 
erratic system behaviors and requiring the use of configuration 
constraints and driver retry logic in operating system software. 
These momentary overload problems are eliminated with the memory 
queued i/o. The target processor can process requests at its own 
pace without interfering with the performance of other parts of 
the system, and in particular without WAILTing or stalling the 
initiating processor. 


A second significant change 15s the potential elimination of 
CPU context swapping caused by the arrival of device interrupts. 
With direct interface i/o, task completion was signalled by 
interrupting the CPU. This either caused a context swap in the 
Operating system, or, if the interrupt were rejected, caused the 
controller to reject further commands and retry the interrupt 
until such time as it was accepted. While still permitting the 
interrupt mechanism to be used, the new i/o also allows for the 
completion of tasks without requiring immediate attention from 
the operating system. 


Finally, the memory queued i/o permits command chaining. 
This, combined with the move of many address calculation chores 
from the CPU to the controller, eliminates the need fora 
dedicated driver process for each device, reduces the need to 
move data back and forth between system buffers and user buffers, 
and facilitates the sharing of i/o resources by dissimilar 
coprocessors and operating systems. 
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FUNCTIONAL REQUIREMENTS 


3.1 General 


3.1.1 Direct Interprocessor I/0 


( Processors supporting the memory queued i/o protocol 
retain support for a small set of direct interprocessor i/o 
commands. These serve primarily initialization, 
identification, and diagnostic functions. The detailed 


descriptions of these commands as presented below presuppose a 
Megabus attachment. It is assumed, however, that they could be 
implemented under any bus architecture that may eventually be 
chosen, while still retaining the same software appearance. 


A processor is permitted to make one of two responses to a 
direct i/o command. It may either accept (ACK) the command or 
reject (NAK) it. Waits are not permitted, despite the fact 
that they are allowed by the Megabus architecture. For the 
Output Control function, there is the further limitation that 
control codes #8000 (initialize) and #4000 (stop i/o) may not 
be NAK*ed. 


3.1.1.1 Output Control Word - Function Code #01 


1 4 rad 
A H O 7 8 7 8 3 


Address MEZ MEZ Target Channel 000001 
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| 4 3 
O > 4 


Data Control Code RFU - MBZ 


3.1.21.1.1 Control Code #8000 - Initialize 


This command causes the target processor to return to 
the same state that it enters after system power on except 
that soft firmware memory, if present, is not cleared. Any 
i/o in process is terminated abruptly and- without 
notification to the target processor. All other fields and 
registers in the target processor are returned to their 
power on state, which is described in detail in Section 
3.1.2 below. QLT‘s, if present, are executed. The 
initialize affects not only the particular channel 
addressed, but also all other channels that reside on the 
same processor. Device resets and/or recalibrates are 
issued where appropriate. This command must always be 
accepted by the target processor. It may not be NAK*‘ed. 


B.1.1.1.2 Control Code #4000 — Stop i/o 


This command causes the target processor to terminate 
any i/o in progress. The current Reguest Header will be 
updated to reflect the status of the in-process i/o, and 
the Rejguest Header will then be engueued to the Completion 
Queue. If there is no i/o currently in process, the Stop 


I/O will be treated as attention interrupt, causing the 


processor to remove a Request Header from Free Request 
Queue O, fill in the appropriate fields, and enqueue it on 
the Completion Queue. This command must always be accepted 
by the target processor. It may not be NAK*‘ed. 


3.21.1.1.3 Control Code #0100 - Start i/o 


This control code signals the target processor to 
search its Request Queue and start processing the highest 
priority (ie., first) Request Block. If no Request Block 
is found on the Reguest Queue, or if there is a Request 
Block from this Request Queue currently being processed, 
this command is ignored. If no valid Initialization 
Control Block has been made known to the target processor, 
bit 15 of the Function Status Word is set. 
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3.1.1.1.4 Control Codes #2000, #1000, #0800, #0400, #0200 

These control codes are supported by processors which 
have loadable firmware. Details are documented in the 
System Maintenance Facility EPS-1. 


3.1.1.2 Output Initialization Control Address —- Function Code 
#O9 


41 4 2 

A H O 7 8 7 8 3 

Address Word Address MSW Target Channel 001001 
11 3 
0 45 4 


Data Word Address LSW 0) RFU -—- MBZ 


This function passes to the target pracessor the 
physical memory address of the Initialization Control Block. 
The function is usually launched by way of the IOLD 
instruction, which initiates an output range (Function Code 
#OB) after the address is sent. If bit 15 of the data word 
(the byte offset) is not zero, bits 13 and 15 of the Header 
Status Word are set and the processors internal copy of the 
ICB and the ICB address are reset as described in Section 
3.1.2. Otherwise, the Status Words are reset as described in 
Section 3.1.2. The format of the Initialization Control 
Block is described in Section 3.1.3.1 below. 


3.1.1.3 Output Initialization Control Range —- Function Code 
#OD 
1 4 2 
A H O 7 8 7 8 3 


Address MEZ MBZ Target Channel 00711014 
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161 
0 9 6 4 
Data Byte Range RFU — MBZ 


This function passes to the target processor the size of 
the Initialization Control Block. The quantity is an 
unsigned 16 bit integer in units of bytes. The value is 
treated as modulo 4 bytes, thus the low order 3 bits of the 
range are ignored by the target processor. The minimum 
permissible range is 40 bytes (#28) and the maximum is 1052 
bytes (#41C). The output range function is usually launched 
by means of the IOLD instruction, which outputs an address, 
followed by a range. Upon receipt of a Function Code #0D, 
the target processor will commence reading of the 
Initialization Control Block. Any of the following 
situations or events will cause the target processor to set 
bit 15 of the Header Status Word and to reset as described in 
Section 3.1.2 its private copies of the ICB and the ICB 


address: 
(. 1. the range is less than #28 (40) or greater than #423 
~~ (1059) 
Tae the processor 15 unable to read the entire 


Initialization Control Block 
B.. the Tditialization- Conthol Block addeess GS ull 


4, the Output Queue address or any of the Free Request 
Queue addresses is null or invalid 


Bits 12, 13, 14, and 15 of Status Word 1 will be set as 
appropriate. Otherwise, Status Words are set as described in 
Section 3.1.2. | 
3.1.1.4 Output Firmware Address ~- Function Code #29 

This function pertains only to processors having 
loadable firmware. Details may be found in System 
Maintenance Facility EPS~t. 
3.1.1.5 Output Firmware Range - Function Code #2D 


This function pertains only to processors having 
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loadable firmware. Details may be found in the System 
Maintenance Facility EPS-1. 


3.1.1.6 Output BRootload Address —- Function Code #39 
TBD 

3.1.1.7 Output Bootload Range - Function Code #3D 
TBD 


3.1.1.8 Input Revision Status - Function Code #04 


Request Cycle 


tJ 


1 (1 
A H O 7 8 7 8 


Address MEZ MBZ Target Channel 000100 
4 14 3 
0 9 0 5 6 4 

Data Request Channel RFU RFU —- MEZ 


Response Cycle 


1 1 c 
A H O 7 8 7 8 3 
Address MBZ MBZ | Request Channel 1001170 
1 4 3 
©) 5 & ‘| 
Data Revision Status RFU —- MBZ | 
This Pane ten returns a 16 bit hardware/firmware 
revision status. The precise meaning is dependent on the 


particular controller and device referenced. 


3.4.1.7 Input Hardware Status Word 1 —- Function Code #18 


9 


HONEYWELL CONFIDENTIAL & PROPRIETARY 


a), 


HONEYWELL INFORMATION 


SPECIFICATION REVISION 
SYSTEMS - 60749952 A 


( 


Request Cycle 


1 (1 2 

A H O 7 8 7 8 3 
Address MBZ MEZ Target Channel 0171000 
1 14 3 
0 9 O > 6 4 


Data Request Channel RFU - MBZ 


Response Cycle 


114 2. 
A H O 7 & 7 8 3 
Address MBZ MEZ Request Channel 100110 
14 3 
0 > 6 4 
Data Status Word 1 Ramware Status 
This command is included primarily for diagnostic 
purposes, and would not be used by the operating system under 
normal circumstances. Status Word 1 15 also returned in the 


Request Body at completion or termination of a function. 
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14.4444 
0123456789012345 


Device Ready : 
Attention 
RFU 


QLT Fault 

Corrected Memory Error 

Reference to Nonexistent Resource 
Bus Parity Error 

Uncorrected Memory Error 


Bits 1 and 12 are reset after Status Word 1 is read, either 
through this function code or by being placed ina Request 
Body. Bits di, 13, 14, and 15 are reset only by the 
initialize and stop i/o functions of the Output Control 
command. 


For controllers which have loadable firmware, the low 
order 16 bits of returned data contain the Ramware Status 


Word. This information is not software visible on current 
machines, and is used only by the System Maintenance 
Facility. Details of the Ramware Status may be found in the 


oystem Maintenance Facility EPS-1. 
3.7.1.10 Input Hardware Status Word 2 - Function Code #1A 


Request Cycle 


41 $1 2 
A H O a oe 7 8 3 

Address MBZ MBZ Target Channel Q0171010 
4 14 | 3 

QO 9 0 2» & a 


Data Request Channel RFU RFU — MBZ 
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Response Cycle 


1 4 2. 
A H O 7 8 7 8 3 
Address MBZ MEZ Request Channel O110710 
161 3 
O 5 6 4 
Data Status Word 2 RFU - MBZ 
This command is included primarily for diagnostic 
purposes, and would not be used by the operating system under 
normal circumstances. Status Word 2 is also returned in the 


Request Body at completion or termination of a function. The 
precise meaning of this status word is device and controller 
Specific. : 
3.1.1.17 Input Function Status - Function Code #1C 


Request Cycle 


£3 12 


-4 
A HO 78 78 


Address MBZ MBZ Target Channel 011100 
11 3: 
6) > & 4 


O 9 
Data Request Channel RFU RFU - MBZ 
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Response Cycle 


1 1 2. 
A H O 7 8 7 8 3 
Address MBZ MBZ Request Channel 101100 
| 1 1 3 
O 5 6 1 
Data Function Status RFU —- MBZ 
This command is included primarily for diagnostic 
purposes, and would not be used by the operating system under 
normal circumstances. The Function Status Word is also 


placed in each Request Body immediately before completion or 


termination of a function. 
1 1 
45 


O12345678 9 


1 1 
) 3 


1 $1 
12 

Request Body Complete 

Segment Fault 

Protection Violation 

Page Fault 

Odd Range Fault : 

RFU 


3.1,.1.12 Input Header Status —- Function Code #/1E 


iim, 
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Request Cycle 


G3 


14 
a H O 7 8 78 


Address MBZ MBEZ Target Channel O1114110 


0 9 
Data Request Channel RFU RFU - MBZ 


Response Cycle 


11 rad 
A H O 7 8 7 8 3 
Address MBZ Reguest Channel 101100 
1°17 3 
O | 5 & 1 
Data Header Status RFU — MBZ 
This command is included primarily for diagnostic 
purposes, and would not be used by the operating system under 
normal circumstances. The Header Status Word is also 


returned in the Request Header immediately before completion 
or termination of a request. 
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=a =a am == 2 a a -_ = aw =a aa == == =a =a =e Deed = = = a= = a a ae nan 


sr es ed 
OtTe@ 384567897071 2345 


Header Complete 
Segment Fault 
Protection Violation 
Paje Fault 

RFU 


Reguest Queue Fault 

Action Queue Fault 

Completion Queue Fault 

Free Reguest Queue Fault 

Invalid Header Fault 

Lost Header Fault 

Address Modularity Fault 

Channel Number Check Failure 
Initialization Control Block Fault 


3.2.1.21.13 Input ID - Function Code #26 
Request Cycle 


1 $1 Fal 
A H O 7 8 7 8 3 


Address MBZ MBZ Target Channel 700110 


164 3 
QO 2» 6 1 


O 9 
Data | Request Channel RFU RFU - MBZ 
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Response Cycle 


11 
A HO 7 8 7 8 3 


Address MBZ MBZ Request Channel 100110 
| 


1-4 
O 5» & 
Data Identification Code RFU — MBZ 


This function returns a 16 bit code which identifies the 
addressed channel. Specific values are assigned for each 
controller and/or device type. These values are documented 
in the MRX Megabus EPS-1. 


3.14.17.13 Input Firmware Load Status - Function Code #3E 


This function is supported only for processors having 
loadable firmware. Details may be found in the System 
Maintenance Facility EPS-1. 


3.1.2 Processor Initial State 


After power up, and after an Initialize command has been 


received (see Section 3.1.1.1.1), the processor’ and = each 
channel on it assume the following state: 


a) The processor‘s internal ICB copies are set with all 
addresses null, and with all ICB Control Flags set to #8000 
(physical address mode). The processors internal ICB address 


fields are set to null. 


b) All error and attention status bits except Hardware 
Status Word 1, bit 11 (GLT status) are reset to zero. QLT 
status and Bit O of each Hardware Status Word 41 (device 
ready) are set as appropriate. 


c) All channels are quiescent, but are prepared to receive 
commands. Those channels that maintain a stored interrupt 
level have that level set to O (no interrupts). 

d) After a full power up (ie., not a power fail restart), 
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all loadable firmware space is cleared. In all other cases, 
loadable firmware is left intact. 


3.1.3 Data Structure Descriptions 


3.1.3.1 Initialization Control Block (ICB) 


Before i/o can commence under the memory queued i/o 
protocol, the target processor must be made aware of the 
location in memory of the necessary data structures. This is 
accomplished by constructing in memory a structure called an 
Initialization Control Block (ICH), which is then made known 
to the target processor. The target processor retains and 
uses a private copy of this structure in its own internal 
RAM. 


If the target processor is functioning in virtual 
address mode, the all the addresses in the RHs and the next 
RB pointer in the RBs are resolved and validated using the 
three I/O Segment Descriptor (IOSD) fields in the ICB. The 
data buffer pointers in the RBs are resolved using the IOSDs 
in the RHs. This address resolution process is described 
more fully in Section 3.1.4 below. If the target processor 
is in physical address mode, the ILI0S5Ds are ignored and the 
Addresses in the data structures are treated as physical 
memory addresses. In either case, however, the sqjueue 
addresses in the ICB itself are physical, not virtual. This 
address mode selection is made via Bit O of the ICB Control 
Flay Word for all addresses except data buffer addresses, and 
in Bit O of the RH Flag Word for data buffer addresses. 
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The format of the ICB is as follows: 
Word Contents 
GO = 3 I/O Sejyment Descriptor 1 
4 - 7 I/O Segment Descriptor 2 


Qs 4 I/O Segment Descriptor 3g - 


‘i2 ICB Control Flags 
13 RFU — MEZ 
14 - 15 Action Queue Header Address 


14 -~ 17 | Request Queue Address 
1G. = 419 Free Request Queue 1 Address 


20 = 24 Free Request Queue 2 Address 


There must be at least one Free Reguest Queue address 
specified; there may be as many as 254. If either the 
Request Queue address or the first FRQ address are null or 
invalid, the Action Queue Header address is mon null and 
invalid, or any subsequent FRQ address is invalid, bit 15 of 
the Header Status Word will be set. This check will take 
place as soon as the ICB is received by the target processor. 


Word 10, the ICB Control Flag Word, has the following 
contents: | 
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1 
Or12345678 907123 45 


Address Mode (1 = physical) 
Multiple Channels (1 = yes) 
Port No. Valid (1 = yes) 


RFU 


Bit O (Address Mode) determines whether addresses in the RBs 
and RHs are to be interpreted as physical or as virtual; see 
Section 3.1.4 for further details. 


Bit di (Multiple Channels) indicates, when set, that the 
Reguest Queue contains RHs for multiple device channels. 
This tells the target processor to check the Local Channel 
Number field in each RH for validity before commencing i/o, 
and to set bit 14 of the Header Status Word if it is invalid. 
If Bit 1 is not set, the target processor will check the 
validity of the Local Channel Number field only if the field 
is non-zero. 


Bit 2 (Port Number Valid) indicates, when set, that the Port 
Number field in the RBs are to be validated and used by the 
target processor. 

3.1.3.2 Request Queue 

TED - If the Request Queue address or any addresses 


encountered in the engueueing or deqgueueing process is found 
to be invalid, bit 7 of the Header Status Word will be set. 
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( 


The format of a Request Header is as follows: 


Word Contents 


: 
: 
: 
12 


Local Destination Channel No. 
Local Port No. 


sal, 
x 


24 —- 27 IOSD 3 
ay = '-34 RFU | | 
beyond Device Specific Extension 
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The Flag Word is set by software and referenced by the 
processor. It contains the following indicators: 


1 
O1723 4567897 01712345 


Address mode (1 = physical) VTE TED 
RFU 


Interrupt Control 


OO = suppress all interrupts 

Ot = interrupt only if CQ 
empty | 

10 = interrupt when header 
complete. 

11 = RFU 


If Bit O (address mode) of the Flag word is set, data buffer 
addresses in the RBs will be treated as physical; otherwise, 
they are translated and validated using the I0SDs in the RH 
as described in Section 3.1.4. 


The Priority Word is a 16 bit unsigned quantity which 
controls the engqueuing and dequeing process, and is used as 
described in the MRX EPS-1 for the CPU queue instructions. 

It is set by software and referenced by the processor. 
Section 3.1.5 below describes the queue management process in 
detail. 


The Pointer to the Next RH is set by software and referenced 
by the processor. It is modified by the processor during the 
dequeuing process. The address can be virtual or physical, 
depending on the address handling mode selected in the ICB. 


The Pointer to the CQ is set by software and referenced by 


the processor. It may be either virtual or physical, 
depending on the address handling mode. 


The Pointer to the Current RB is set by software, and 15 

managed by the processor as is works its way through the RB 

list. It is always set to point to the current active RE. 

At the normal completion of a sequence of REs, it will point 
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to the last RB. If termination is abnormal, it will point to 
the RB that was active at the time of termination. It may be 
either virtual or physical, depending on the address handling 
mode. 


The Pointer to the First RB is set by software and referenced 
by the processor. It may be either virtual or physical, 
depending on the address handling mode. 


The Header Status Word is set by the processor to reflect any 
of a variety of faults that can occur during processing of a 


RQ. Immediately before termination or completion of 
processing on the header, Bit O 1s set to indicate that 
processing is complete. Section 3.1.1.12 describes the 


Header Status Word in detail. 


The Ring Number is used for read and write access checking of 
data buffers when operating in virtual memory mode. Section 
3.1.4 describes this memory management in detail. 


The Local Destination Channel Number specifies the physical 
i/o channel on the local bus to which the RH 15 directed. It 
is set by software and referenced by the processor. If Bit 1 
of the ICB Control Word (see Section 3.1.3.1) is set, this 
channel is can be used by the processor to chose from among 
possible device channels, and the channel will be checked for 
validity. This permits reguests for multiple physical 
channels to be placed in a single RQ. If Bit 1 of the ICB 
Control Word is zero, the channel number will be ignored if 
it is zero, otherwise, it will be checked for validity. If 
Bit 2 of the ICB Control Word is set, indicating the presence 
of a Local Port Number, then only the portion of the Local 
Destination Channel Number which pertains to the controller 
base channel will be validated. Validity errors result in 
abnormal termination and the setting of Bit 14 in the Header 
Status Word. 


The Local Port Number specifies a controller port number, and 
is valid only if Bit 2 of the ICB Control Word (see Section 
3.1.3.1) i565 set. It is set by software and used by the 
processor to chose from among ports on a controller. It is of 
particular use in situations in which multiple controllers 
are sharing ‘queues. This subject is presented in detail in 
Section 4 below. 2 


The Local Source Channel Number is set by the software and 
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referenced by the processor. Typically, ait will be a CPU 
channel number. Appended to it is a level number. These 
fields are used by the processor when sending an interrupt. 
It is not verified by the processor. 


The Active Channel Number is set by the processor to reflect 
the physical channel that is acting on this RH. 


The I0SDs in the RH are used only in the resolution of the 
Data Buffer Pointers in the REs. The detailed contents of 
the ITOSDs is discussed in Section 3.1.4.1 below. 


The format of a Request Body (RB) is as follows: 


Word Contents 


a 
3 Hardware Status Word 1 

: 
> Function Status Word 

i= AS Buffer Descriptor 1 


TBD - descriptions, esp Function Status. 
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The format of a Buffer Descriptor is as follows: 


Word Contents 


O-- 1 Pointer to Data Buffer 
: 
3 


Residual Range 


3.21.23.3 Action Queue 


The Action Queue serves as a repository for Reguest 

Blocks which are currently in use. It is an optional 

structure. If an Action Queue is present, each Request Block 

will be enqueued to it as soon as it is detached froma 

Request Queue or Free Reyguest Queue, and will remain there 

until such time as it is placed on a Completion Queue. This 

feature assists system resilience and problem diagnosis by 

— eliminating, to the extent possible, the likelihood that a 
( request will “disappear" due to hardware malfunction. 


The presence of an Action Queue is indicated toa 
processor by providing a valid physical address in the Action 
Queue Header Address field of the ICB. If the address is 
null, the Action Queue processing step is skipped. If the 
Action Queue address or any addresses encountered in the 
enqueueing or dequeueing process 1s found to be invalid, bit 
8 of the Header Status Word will be set. 


3.1.3.4 Completion Queue 
TBD - If the Completion Queue address or any address 
encountered in the enjueueing or dequeueing process is found 


to be invalid, bit 9 of the Header Status Word will be set. 


3.1.3.5 Free Request Queue 

TBD - If a Free Request Queue address or any address 
encountered in the engueueing or dequeueing process is found 
to be invalid, bit 10 of the Header Status Word will be set. 


3.1.4 Memory Address Handling 
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3.1.4.1 Virtual Address Handling 


Processors may function in either virtual or physical 
address handling modes. In the first of these, all pointers 
are treated as physical memory addresses, and are subject 
neither to address translation nor to access checking. In 
this mode, 1/0 Segment Descriptors (I1OSDs) are ignored. 


In the normal situation, processors will be functioning 
in virtual address mode. In this mode, both translation and 
access checking will be done on all pointers. The method 
closely parallels that described in the MRX EPS-1. The 
mechanism to change address handling mode is detailed in 
sections 3.1.3.1 and 3.1.3.2 above. 


Physical or virtual address handling may each be chosen 
two points. Bit O of the ICE Control Flag Word determines 
whether addresses in the RHs and the next RB Pointer in the 
RRs are to be treated as physical or virtual. If virtual 
addressing is selected, the three IO0SDs in the ICB are used 
for mapping and address control. Bit O of the RH Flag Word 
determines whether data buffer addresses in the RHs are to be 
treated as virtual; if so, the three I10SDs in the RH are 
used. | 


Note that the pointers in the ICB will always be 
physical, regardless of the processor address handling mode. 


The format of an IOSD is as follows: 


PAGE NO. (12:19) 
Lacs 
fe fe 


The individual I0SD fields have the meanings listed below. 
Those fields marked RSU are iynored: | 
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Ve- if set to 1, the IOSD is valid. If set to 0, the IOSD 
is ignored. 


PR - RSU (privilege indicator) 


ST - RSU (segment type) 


P - if set to 1, segment is paged. 
E - RSU (execute-permit indicator) 
R - if set to 1, and if the Ring Number in the RH is less 


than or equal to Re, data may be read from this segment. 
If an unpermitted attempt to do s0 occurs, Bit 2 of the 
Function Status Word will be set and processing of the RH 
is terminated. Note that this corresponds to a data write. 


W o- if set to 1, and if the Ring Number in the RH is less 
than or egual to R1, data may be written into this segment. 
lf an unpermitted attempt to do so occurs, Bit 2 of the 
Function Status Word will be set and processing of the RH 
Will terminate. Note that this corresponds to a data read. 


R1 - highest (ie., least privileged) ring number into which 
a memory write (data read) is allowed 


R2 - highest (ie., least privileged) ring number from which 
a memory read (data write) is allowed 


R3 - RSU (highest call bracket ring number) 
C - RSU (compatibility indicator) 


G - RSU (gate segment indicator) 


Segment Size - if P = QO, this is the size, in pages, of 
this segment. lf P = 1, this is the size of the page 
table. | 

Segment Number - valid segment number for this segment. 


Pajge Number ~- if P = 0, this field defines the start of the 
non-paged segment in main memory. If P = 1, this and the 
Offset field define the start in main memory of the Page 
Table for this segment. 3 
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Offset - if P = 0, this field must be zero. If P = 4, this 
field and the Page Number define the start of the Page 
Table for this segment. : 


Fields R, W, R1, and R2 are meaningful only for LQSDs in the 
RHs. For the I0SDs in the ICKB, they are ignored. 


Segments may be either paged or unpaged. If paged, the LOSD 


defines a Page Table which contain two word Page Descriptor 
entries of the following form: 


1 2. 3 4 > & 7 8 > 


PAGE FRAME ADDRESS (0:41) 
PAGE FRAME ADDRESS (12:19) RSU 


Within a Page Descriptor (PD) entry, fields have the 
following forms: 


V- ROU (valid indicator) 


| U—- PD used indicator. This field is set by the processor 
' using memory lock (read-modify-write) whenever data is 
» written from or read into the memory defined by the PD. 


M - PD modified indicator. This field is set by the 
processor using memory lock (read-modify-write) whenever 
data is read into the memory defined by the PD. 


i; Page Frame Address - 20 bit physical page frame address 
TED ~ address calculation description 
3.1.4.2 Structure Address Constraints 


Two important constraints are applied to the addresses 
of the RBs and RHs. First, they must be located on modulo 16 
word boundaries. If a pointer to one of these structures is 
encountered for which the low order 4 bits of the word 
address are not zeroes, the MQIO operation in progress will 
terminate abnormally and bit 13 of the Header Status will be 
set. If the offending address is an RB pointer, the RH will 
be engueued on the Completion Queue. If it is an RH pointer 
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that violates the rule, the controller will attempt to signal 
the fault as if it were an attention item, ie., it will 
attempt to dequeue a RH from FRGO and posting it on the 
Completion Queue. If the controller cannot find a legal RH 
on FRGO, it will set bits 12 and 13 of the Header Status. 


The second limitation on RBs and RHs is that they not 
span page boundaries. Violation of this rule may be 
undetected by the target processor. Only the address of the 
first word of either of these two structures will be checked 
for validity. 


These two restrictions are placed in order to optimize 
processing speed and to insure the integrity of pointers. 


3.1.4.3 Data Address Constraints 


In order to maximize performance, it is a requirement 
that all data buffer addresses be word aligned. In keeping 
with this restriction, the Buffer Pointers in the REs are 
word pointers. In addition, within an RB, all ranges except 
the last one in an set (ie., the set of input buffer 
descriptors or the set of output buffer descriptors) must be 
even. If this rule is violated, bit 4 of the Function Status 
Word will be set and the data transfer will terminate 
abnormally. The last range in aset may be odd; in this 
case, the data transfer will overrun one byte. 


3.1.5 Handling of the Queuing Process 

The Request Queue, the Action Queue, the Completion Queue, 
and the Free Request Queues all are or can be shared by both 
software and hardware, thus the handling of these queues must 
be consistent and must conform to the queue management 
standards presented in Section 3.15 of the MRX EPS-1. Briefly, 
each queue fas a queue header, which contains a pointer to the 
first item in the queue, a pointer to the last item in the 
queue, and a lock word. For DPS& and DPS&+ software, it is 
assumed that these will be managed using the CPU queue 
instructions QOH, QOT, DGH, DQA, SQH, and SGA. 


For non-DPS& CPUs and for other processors, the following 
sequence must be observed. To gain access to the queue, the 
lock word must first be checked and set. This must be done 
using a read-modify-write cycle (lock cycle) to insure that 
only one user at a time can have access to the queue. If a 
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processor finds that the queue lock word bit 15 is set, it must 
wait a short time, then try again to get access to the queue. 
If bit 15 is not set, it must be set, and the processor'‘s 
channel number must be placed in bits O through &. 


Having gained access to the gueue, a processor may search 
by following the chain of frame pointers. To dejueue an item, 
a processor sets the frame pointer from the prior queue frame 
or header to the value contained in the next frame pointer of 
the frame to be deqgueued. To engjueue an item, the next frame 
pointer in the prior queue frame is set to point to the new 
frame, and the next frame pointer in the new frame is set to 
the value previously in the prior frame. Note that processors 
using the MQIO always “queue on tail", that is, an item being 
linked to a queue is placed in the queue after all items having 
lesser or equal priority word values, and before any items 
having greater priority word values. 


321.26 I/O Scenarios 


TBD - illustrative examples 


3.2 Device and Processor Specific I/0 


3.2.1 Magnetic Peripherals 
TBD - collection of statistics 
3.2.2. Communications 

TED 

3.2.3 Cluster Controller 


The format of the device specific extension to the RB is 
as follows: 
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Word Contents 


: 
: 
: 
: 
38 - 47 
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LV 


SOFTWARE IMPLEMENTATION AND APPLICATION 


4.1 Sharing of Queues 


In the simplest form of MQIO, each peripheral channel has 
its own set of queues. There 1i8, however, nothing in the 
design of MQIO that precludes gueue sharing. The queue locking 
mechanism employed by the DPS-& yjueue instructions is used by 
all M@GIOQ controllers to maintain sgueue consistency, and in 
fact, it is anticipated that Completion Queue sharing will be 
the norm for most system software. | 


Sharing of Request Queues can be done in three different 
ways, all supported by the MGIO protocol. In the first scheme, 
each Request Header contains the peripheral channel number of 
its target. Controllers associated with the gueue each scan 
the queue looking for Request Headers that pertain to them; 
when a controller finds a Request Header that belongs to it, 
the Header is handled as described above. This queue sharing 
method may offer some advantages for the driver software, but 
conceptually it ais little different from having independent 
queues. : 


With the second method, Request Headers do not contain 
target peripheral channel numbers. Thus, any controller on the 
queue will seize and act upon any Request Header that it finds. 
This method is appropriate when all channels associated with 
the queue can perform the same functions. This scheme is used 
in the multiple cluster controller case described below. A 
variant of this scheme permits some Request Headers to he 
marked with channel numbers, and some to be left unmarked, thus 
allowing the driver to explicitly direct some i/o activities to 
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particular channels while permitting others to be handled by 
any available controller. 


With the third method, certain Request Headers are marked 
with a device port number, but not with a channel number. 
Section 1V below presents the details of this approach. 


4.1.1 Multiple Cluster Controllers Sharing Queues 


For resiliency and throughput reasons it is convenient 
to have multiple paths between nodes of a cluster. To do 
this, the following configuration might be used: 


System A 
Cluster Cluster 
Controller Controller 
4 
Star Star 
Coupler Coupler 
4 c. 


Cluster Cluster 


Controller 


Controller 


1 2 
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eT 


If on each system both cluster controllers share the same 
queues, throughput could be nearly double that of a single 


of failure during data transfer, the software would have to 
unwind the failed transaction and disable the failed cluster 


i Cluster connection. Except in the failure case, the 
' existence of more than one cluster controller could be 
( completely transparent to the driver software. In the case 
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link. For other failures, it would suffice merely to disable 
the failed path. For even greater throughput and/or fault 
tolerance, three or more cluster links could be established, 
all sharing the same queues. Except for collisions at the 
queue locks and on the system busses, there should be no 
friction between multiple cluster links configured in this 
fashion. 


4.1.2 Multiple Disk Controllers with Multi-port Drives 


Consider the following configuration: 


System Bus 


=~ tem—~ OO BReorAaan 
oer ORBerg7 8 gO 


Db 
8, 


This combination offers both throughput and resiliency 
advantages over a similar single controller configuration, 
but with current i/o techniques, a penalty would be paid in 
the driver software. This penalty arises primarily because 
each view of each disk is identified by a separate channel 
number. By contrast, under MGIO, Request Headers could be 
marked with port numbers (as the term 15s used here, port 
number refers to the controller‘s port number, not the disk 
device‘s port number) but not channel numbers. Both 
controllers could share the same set of queues, and either 
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controller could handle i/o to a particular dual port disk. 
This behavior would be largely invisible to the drivers. If 
a controller failure were detected, recovery software would 
have to unwind the failed operation, which could be found on 


the Action Queue, and disable the failed controller. 
Otherwise, physical i/o processing would continue as before, 
although at a slower pace. If multi-port disks were 


available, this scheme could be expanded with more 
controllers. | 


Queues under MGIO may be at the physical channel level 
as well as at the controller level. This leads to useful 
variant of the shared gueue technique which could be applied 
to the following configuration: 


System Bus 


Disk 1 


Port 1 
Disk 2 


5S Deere O seaoa NY 
=~ DTeew Om ergaan 


Db 
wo 


Here, controllers "A" and "B" could share a set of queues for 

disk 1, but disks @2@ and @3 could be driven from separate 

queues. Alternatively, all three disks could share the same 
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queue set, but the Request Headers for disks 2 and 3 could be 
tagged with channel numbers, while the Headers for disk 1 
could be tagged only with port numbers. Either way, access 
to disk 1 (perhaps the system disk) would be substantially 
immune from a single controller failure. 


2 Configuration Considerations 


It is generally recognized that the configuration process 
in future operating systems must be much less visible to the 
user than it is today under VS1. This, combined with the new 
dimension of configuration possibilities presented with MQIO, 
requires substantial help at the hardware level. It must be 
Possible on new MGIO controllers to answer the following 
questions: 


ds Which peripherals are connected to which controllers? 
In many cases, this is trivial to determine, but there 
currently are some 4, 6, and 8 port disk controllers for 
which mere knowledge of device id‘s may be insufficient. 


2. Which channels on two controllers correspond to the same 
dual ported device? It is of course possible to determine 
this by writing to a device from one port, then reading it 
from the other. This is quite inconvenient, anda better 
method should be sought. 


It would be best if a uniform solution suitable for all MQIO 
controllers could be found and implemented. A good place to 
start would be to require all such controllers to support a 
supervisory channel separate from the peripheral channels. 
This supervisory channel could supply controller-wide 
configuration information as well as other special diagnostic 
and resiliency functions. 
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Appendix I 
Reference Documentation 
60149789, MRX System and Central Subsystem EPS-1, Revision D 


60149831, DPS& 32 Bit Product System Management Facility EPS-1, 
Revision E— 


60149832, MRX Megabus EPS-1, Revision E 
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Appendix IL 


Abbreviations 


~ AQ - Action Queue —- see Section 3.1.3.3 
CPU -~ Central Processor Unit 
CQ - Completion Queue —- see Section 3.1.9.4 
DIIO - Direct Interface I1/0 
Ero ~ Engineering Product Specification, Part 1 
FRQ - Free Request Queue - see Section 3.1.3.5 
I/O - Input/Output 
LOSD ~ I/O Segment Descriptor - see Section 3.1.4 
ICB - Initialization Control Block ~- see Section 3.1.3.1 
MBN -~ Must Be Null Address 
MBZ - Must Be Zero 
MQIO ~ Memory Queued I/O Protocol 
NULL - An Address of All Zeroes 
RAM - Random Access Memory 
RFU - Reserved for Future Use 
RB - Request Hody 
RH ~- Request Header 
a RQ - Request Queue —- see Section 3.1.3.2 
( RSU - Reserved for Software Use 
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